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Abstract 


We study online learning where a decision maker interacts with Nature with the objective of max- 
imizing her long-term average reward subject to some sample path average constraints. We define 
the reward-in-hindsight as the highest reward the decision maker could have achieved, while sat- 
isfying the constraints, had she known Nature’s choices in advance. We show that in general the 
reward-in-hindsight is not attainable. The convex hull of the reward-in-hindsight function is, how- 
ever, attainable. For the important case of a single constraint, the convex hull turns out to be the 
highest attainable function. Using a calibrated forecasting rule, we provide an explicit strategy 
that attains this convex hull. We also measure the performance of heuristic methods based on 
non-calibrated forecasters in experiments involving a CPU power management problem. 


Keywords: online learning, calibration, regret minimization, approachability 


1. Introduction 


We consider a repeated game from the viewpoint of a decision maker (player P1) who plays against 
Nature (player P2). The opponent (Nature) is “arbitrary” in the sense that player P1 has no pre- 
diction, statistical or strategic, of the opponent’s choice of actions. This setting was considered 
by Hannan (1957), in the context of repeated matrix games. Hannan introduced the Bayes utility 
with respect to the current empirical distribution of the opponent’s actions, as a performance goal 
for adaptive play. This quantity, defined as the highest average reward that player P1 could have 
achieved, in hindsight, by playing some fixed action against the observed action sequence of player 
P2. Player P1’s regret is defined as the difference between the highest average reward-in-hindsight 
that player P1 could have hypothetically achieved, and the actual average reward obtained by player 
P1. It was established in Hannan (1957) that there exist strategies whose regret converges to zero as 
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the number of stages increases, even in the absence of any prior knowledge on the strategy of player 
P2. For recent advances on online learning, see Cesa-Bianchi and Lugosi (2006). 

In this paper we consider regret minimization under sample-path constraints. That is, in addition 
to maximizing the average reward, or more precisely, minimizing the regret, the decision maker has 
some side constraints that need to be satisfied on the average. In particular, for every joint action 
of the players, there is an additional penalty vector that is accumulated by the decision maker. The 
decision maker has a predefined set in the space of penalty vectors, which represents the accept- 
able tradeoffs between the different components of the penalty vector. An important special case 
arises when the decision maker wishes to keep some constrained resource below a certain threshold. 
Consider, for example, a wireless communication system where the decision maker can adjust the 
transmission power to improve the probability that a message is received successfully. Of course, 
the decision maker does not know a priori how much power will be needed (this depends on the 
behavior of other users, the channel conditions, etc.). Still, a decision maker is usually interested in 
both the rate of successful transmissions, and in the average power consumption. In an often consid- 
ered variation of this problem, the decision maker wishes to maximize the transmission rate, while 
keeping the average power consumption below some predefined threshold. We refer the reader to 
Mannor and Shimkin (2004) and references therein for a discussion of constrained average cost 
stochastic games and to Altman (1999) for constrained Markov decision problems. We note that the 
reward and the penalty are not treated the same; otherwise they could have been combined into a 
single scalar value, resulting in a much simpler problem. 

The paper is organized as follows. In Section 2, we present formally the basic model, and 
provide a result that relates attainability with the value of the game. In Section 3, we provide 
an example where the reward-in-hindsight cannot be attained. In light of this negative result, in 
Section 4 we define the closed convex hull of the reward-in-hindsight, and show that it is attainable. 
Furthermore, in Section 5, we show that when there is a single constraint, this is the maximal 
attainable objective. In Section 6, we provide a simple strategy, based on calibrated forecasting, 
that attains the closed convex hull. Section 7 presents heuristic algorithms derived from an online 
forecaster, while incorporating strictly enforced constraints. The application of the algorithms of 
Section 7 to a power management domain is presented in Section 8. We finally conclude in Section 
9 with some open questions and directions for future research. 


2. Problem Definition 


We consider a repeated game against Nature, in which a decision maker tries to maximize her 
reward, while satisfying some constraints on certain time-averages. The underlying stage game is 
a game with two players: P1 (the decision maker of interest) and P2 (who represents Nature and is 
assumed arbitrary). For our purposes, we only need to define rewards and constraints for P1. 

A constrained game with respect to a set T is defined by a tuple (A,B,R,C,T) where: 


1. A is the set of actions of P1; we will assume A = {1,2,...,|A]}. 
2. Bis the set of actions of P2; we will assume B = {1,2,...,|B]}. 


3. R is an |A| x |B| matrix where the entry R(a,b) denotes the expected reward obtained by 
P1, when P1 plays action a € A and P2 action b € B. The actual rewards obtained at each 
play of actions a and b are assumed to be IID random variables, with finite second moments, 
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distributed according to a probability law Pre(-|a,b). Furthermore, the reward streams for 
different pairs (a,b) are statistically independent. 


4. C is an |A| x |B| matrix, where the entry C(a,b) denotes the expected d-dimensional penalty 
vector incurred by P1, when P1 plays action a € A and P2 action b € B. The actual penalty 
vectors obtained at each play of actions a and b are assumed to be IID random variables, with 
finite second moments, distributed according to a probability law Prc(-|a,b). Furthermore, 
the penalty vector streams for different pairs (a,b) are statistically independent. 


5. T is a set in R within which we wish the average of the penalty vectors to lie. We assume 
that T is convex and closed. Since the entries of C are bounded, we will also assume, without 
loss of generality, that T is bounded. 


The game is played in stages. At each stage t, P1 and P2 simultaneously choose actions a; € A 
and b; € B, respectively. Player P1 obtains a reward r,, distributed according to Prr(-|a;,b;), anda 
penalty c+, distributed according to Prc(-|a;,b;). We define P1’s average reward by time ż to be 


n lš 
ñ =-} rm, (1) 
t t=1 
and P1’s average penalty vector by time f to be 
1 t 
& ==) cr. (2) 
t t=1 


A strategy for P1 (resp. P2) is a mapping from the set of all possible past histories to the set of 
mixed actions on A (resp. B), which prescribes the (mixed) action of that player at each time f, as a 
function of the history in the first t — 1 stages. Loosely, P1’s goal is to maximize the average reward 
while having the average penalty vector converge to T, pathwise: 


limsupdist(ĉ,, T) — 0, a.s., (3) 


t— oo 


where dist(-) is the point-to-set Euclidean distance, that is, dist(x, T) = infyer ||y — x||2, and the 
probability measure is the one induced by the policy of P1, the policy of P2, and the randomness in 
the rewards and penalties. 

We will often consider the important special case where T = {c € Rf : c < co}, for some given 
co € RÍ, with the inequality interpreted component-wise. We simply call such a game a constrained 
game with respect to (a vector) co. For that special case, the requirement (3) is equivalent to: 


limsup¢é; < co, a.s.. 


t— oo 


For a set D, we will use the notation A(D) to denote the set of all probability measures on D. If 
D is finite, we will identify A(D) with the set of probability vectors of the same size as D. If D is a 
subset of Euclidean space, we will assume that it is endowed with the Borel o-field. 
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2.1 Reward-in-hindsight 


We define g, € A(B) as the empirical distribution of P2’s actions by time ¢, that is, 


1 t 
â (b) = L l=} DEB. (4) 
t=1 


If P1 knew in advance that g, will equal q, and if P1 were restricted to using a fixed action, then 
P1 would pick an optimal response (generally a mixed action) to the mixed action q, subject to the 
constraints specified by T. In particular, P1 would solve the convex program! 


Ie 2 AREA (5) 
St. 2 Piaatb)clap) ET. 


By playing a p that solves this convex program, P1 would meet the constraints (up to small fluctua- 
tions that are a result of the randomness and the finiteness of t), and would obtain the maximal aver- 
age reward. We are thus led to define P1’s reward-in-hindsight, which we denote by r* : A(B) — R, 
as the optimal objective value in the program (5), as a function of q. The function r* is often referred 
to as the Bayes envelope. 

For the special case of a constrained game with respect to a vector co, the convex constraint 
Yap P(a)q(b)C(a,b) €T is replaced by X4 p p(a)q(b)C(a,b) < co (the inequality is to be interpreted 
component-wise). 

The following examples show some of the properties of the Bayes envelope. Consider a 2 x 2 
constrained game with respect to a scalar cg specified by: 


where each entry (pair) corresponds to (R(a,b),C(a,b)) for a pair of actions a and b. (Here a 
and b correspond to a choice of row and column, respectively.) Suppose first that co = 1. In that 
case the constraint does not play a part in the problem, and we are dealing with a version of the 
matching pennies game. So, if we identify q with the frequency of the first action, we have that 
r*(q) = max(q,1 — q). Suppose now that co = 1/2. In this case, it is not difficult to show that 
r*(q) = 1/2, since P1 cannot take advantage of any deviation from q = 1/2 while satisfying the 
constraint. 

The next example involves a game where P2’s action does not affect the constraints; such games 
are further discussed in Section 4.1. Consider a 2 x 2 constrained game with respect to a scalar co, 
specified by: 





1. If T is a polyhedron (specified by finitely many linear inequalities), then the optimization problem is a linear program. 
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where each entry (pair) corresponds to (R(a,b),C(a,b)) for a pair of actions a and b. We identify 
q with the frequency of the second action of P2 as before. Suppose first that co = 1. As before, 
the constraint has no effect and r*(q) = max(g, 1 — q). Suppose now that cp = 1/2. It is not hard 
to show that in this case r*(q) = max(q, 1/2). Finally, if co = 0, P1 is forced to choose the second 
action; in this case, r*(q) = q. The monotonicity of r*(q) in co is to be expected since the lower co 
is, the more stringent the constraint in Eq. (5). 


2.2 The Objective 


Formally, our goal is to attain a function r in the sense of the following definition. Naturally, the 
higher the function r, the better. 


Definition 1 A function r : A(B) + R is attainable by P1 in a constrained game with respect to a 
set T if there exists a strategy © of P1 such that for every strategy p of P2: 


(i) liminf,_,..(7;-—r(G+)) >0, a.s., and 
(ii) limsup,_,..dist(¢,,T) -0, a.s., 
where the almost sure convergence is with respect to the probability measure induced by © and p. 
In constrained games with respect to a vector co we can replace (ii) in the definition with 


limsupĉ, < co, a.s. 


t—00 


2.3 The Value of the Game 


In this section, we consider the attainability of a constant function r : A(B) + R, that is, r(q) = a, 
for all q. We will establish that attainability is equivalent to having a < v, where v is a naturally 
defined “value of the constrained game.” 

We first introduce the assumption that P1 is always able to satisfy the constraint. 


Assumption 1 For every mixed action q € A(B) of P2, there exists a mixed action p € A(A) of P1, 
such that: 


2 pla)a(b)C(a,b) ET. (6) 


For constrained games with respect to a vector co, the condition (6) reduces to the inequality 
Lap P(a)q(b)C(a,b) < co. 

If Assumption 1 is not satisfied, then P2 can choose a q such that for every (mixed) action of P1, 
the constraint is violated in expectation. By repeatedly playing this q, P1’s average penalty vector 
will be outside T, and the objectives of P1 will be impossible to meet. 

The following result deals with the attainability of the value, v, of an average reward repeated 
constrained game, defined by 


v= inf sup $ p(a)q(b)R(a,b). (7) 
qEA(B) pEA(A):Ea p p(a)g(b)C(a,b) ET a,b 


The existence of a strategy for P1 that attains the value was proven in Shimkin (1994) in the broader 
context of stochastic games. 
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Proposition 2 Suppose that Assumption I holds. Then, 


(i) P1 has a strategy that guarantees that the constant function r(q) = v is attained with respect 
toT. 


(ii) For every number v' > v there exists 6 > 0 such that P2 has a strategy that guarantees that 
either liminf;...7; < v’ — 6 or limsup,_,,, dist(¢,,T) > ò, almost surely. (In particular, the 
constant function v' is not attainable.) 


Proof The proof relies on Blackwell’s approachability theory (Blackwell, 1956a). We construct a 
nested family of convex sets in R+! defined by Sa = {(r,c) € Rx R4:r > a,c € T}. Obviously, 
Sa C Sp for & > B. Consider the vector-valued game in R?! associated with the constrained game. 
In this game, P1’s vector-valued payoff at time t is the d + 1 dimensional vector m; = (r+, c+) and P1’s 
average vector-valued payoff is 7, = (7;,¢;). Since Sq, is convex, it follows from approachability the- 
ory for convex sets (Blackwell, 1956a) that each Sg is either approachable? or excludable.> If Sx is 
approachable, then Sg is approachable for every B < a. We define vo = sup{f | Sg is approachable}. 
It follows that Sy, is approachable (as the limit of approachable sets; see Spinat, 2002). By Black- 
well’s theorem, for every g € A(B), an approachable convex set must intersect the set of feasible 
payoff vectors when P2 plays q. Using this fact, it is easily shown that vo equals v, as defined by 
Eq. (7), and part (i) follows. Part (ii) follows because a convex set which is not approachable is 
excludable. a 


Note that part (ii) of the proposition implies that, essentially, v is the highest average reward P1 
can attain while satisfying the constraints, if P2 plays an adversarial strategy. By comparing Eq. (7) 
with Eq. (5), we see that v = inf, r“ (q). On the other hand, if P2 does not play adversarially, P1 may 
be able to do better, perhaps attaining r*(q). Our subsequent results address the question whether 
this is indeed the case. 


Remark 3 In general, the infimum and supremum in (7) cannot be interchanged. This is because 
the set of feasible p in the inner maximization depends on the value of q. Moreover, it can be shown 
that the set of (p,q) pairs that satisfy the constraint Yq p(a)q(b)C(a,b) E€ T is not necessarily 
convex. 





2. A set X is approachable if there exists a strategy for the agent such that for every € > 0, there exists an integer N such 
that, for every opponent strategy: 


1 n 
Pr (ax (: mx) > £ for some n > n) <E. 
nE 


3. A set X is excludable if there exists a strategy for the opponent such that there exists 6 > 0 such that for every € > 0, 
there exists an integer N such that, for every agent strategy: 


1 n 
Pr (ax (} mx] > 6 for all n > n) >1-—e. 
n iZi 
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2.4 Related Works 


Notwithstanding the apparent similarity, the problem that we consider is not an instance of online 
convex optimization (Zinkevich, 2003; Hazan and Megiddo, 2007). In the latter setting, there is a 
convex feasible domain F C R”, and an arbitrary sequence of convex functions f; : F — R. At 
every step ft, the decision maker picks x, E€ F based on the past history, without knowledge of the 
future functions /;, and with the objective of minimizing the regret 


T F 
L F) = min $, f0). 
t=1 YEF p=] 


An analogy with our setting might be possible, by identifying x; and f; with a; and b;, respectively, 
and by somehow relating the feasibility constraints described by ¥ to our constraints. However, 
this attempt seems to run into some fundamental obstacles. In particular, in our setting, feasibility 
is affected by the opponent’s actions, whereas in online convex optimization, the feasible domain 
F is fixed for all time steps. For this reason, we do not see a way to reduce the problem of online 
learning with constraints to an online convex optimization problem, and given the results below, it 
is unlikely that such a reduction is possible. 


3. Reward-in-Hindsight Is Not Attainable 


As it turns out, the reward-in-hindsight cannot be attained in general. This is demonstrated by the 
following simple 2 x 2 matrix game, with just a single constraint. 
Consider a 2 x 2 constrained game specified by: 


CE. 


where each entry (pair) corresponds to (R(a,b),C(a,b)) for a pair of actions a and b. At a typical 
stage, P1 chooses a row, and P2 chooses a column. We set co = 0. Let q denote the frequency 
with which P2 chooses the second column. The reward of the first row dominates the reward of 
the second one, so if the constraint can be satisfied, P1 would prefer to choose the first row. This 
can be done as long as 0 < q < 1/2, in which case r*(q) = 1. For 1/2 < q < 1, player P1 needs 
to optimize the reward subject to the constraint. Given a specific q, P1 will try to choose a mixed 
action that satisfies the constraint (on the average) while maximizing the reward. If we let œ denote 
the frequency of choosing the first row, we see that the reward and penalty are: 


r(a,g) =a—(1—aq)q, c(a,q) = 20g—1, 


respectively. We observe that for every q, r() and c(a) are monotonically increasing functions of 
a. As a result, P1 will choose the maximal © that satisfies c(a@) < 0, which is a(g) = 1/2q, and the 
optimal reward is 1/2 + 1/2q — q. We conclude that the reward-in-hindsight is: 


1, if0<q<1/2, 
rq@=4, 1 1 
-+—— if1/2<qg<l. 


The graph of r*(q) is the solid line in Figure 1. 
We now claim that P2 can make sure that P1 does not attain r*. 
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2x2 R-C game 

















Figure 1: The reward-in-hindsight of the constrained game. Here, r*(q) is the solid line, and the 
dotted line connects the two extreme values, for q = 0 and q = 1. 


Proposition 4 If co = 0, then there exists a strategy for P2 such that r* cannot be attained. 


Proof Suppose that the opponent, P2, plays according to the following strategy. Initialize a counter 
k = 1. Let â, be the empirical frequency with which P1 chooses the first row during the first t time 
steps. Similarly, let ĝ, be the empirical frequency with which P2 chooses the second column during 
the first ¢ time steps. 


1. While k = 1 or &—; > 3/4, P2 chooses the second column, and k is incremented by 1. 


2. For the next k times, P2 chooses the first column. Then, reset the counter k to , and go back 
to Step 1. 


We now show that if 


limsup¢é,<0, a.s., 


t—00 


then a strict inequality holds for the regret: 


liminf(#; —r*(â)) <0, a.s. 


t—0o 


Suppose that Step 2 is entered only a finite number of times. Then, after some finite time, P2 keeps 
choosing the second column, and ĝ, converges to 1. For P1 to satisfy the constraint lim sup, ĉr < 
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0, we must have lima, < 1/2. But then, the condition &,_; > 3/4 will be eventually violated. This 
shows that Step 2 is entered an infinite number of times. In particular, there exist infinite sequences 
t; and t such that t; < t < ti+1 and (i) if t; < t < t , P2 chooses the second column (Step 1); (ii) if 
ti <t < ti+1, P2 chooses the first column (Step 2). 

Note that Steps 1 and 2 last for an equal number of time steps. Thus, we have g;, = 1/2, and 
r*(4;,) = 1, for all i. Furthermore, t;}1 — ti < tj, or t; > t;41/2. Note that & < 3/4, because otherwise 
P2 would still be in Step 1 at time ¢/+ 1. Thus, during the first f;+; time steps, P1 has played the 
first row at most 


31/44 (tigi =t) = titi —t)/4 < Ttiz1/8 


times. Due to the values of the reward matrix, we have limsup,_,., 7; < limsup,_,../;,. In particular, 
we have 7;,,, < 7/8, and lim inf;_,.. (7; — r*(G)) < 7/8 -—1 <0. a 


Intuitively, the strategy that was described above allows P2 to force P1 to move, back and forth, 
between the extreme points (q = 0 and q = 1) that are linked by the dotted line in Figure 1. Since 
r*(q) is not convex, and since the dotted line is strictly below r*(q) for q = 1/2, this strategy 
precludes P1 from attaining r*(q). We note that the choice of co is critical in this example. With 
other choices of cp (for example, co = — 1), the reward-in-hindsight may be attainable. 


4. Attainability of the Convex Hull 


Since the reward-in-hindsight is not attainable in general, we have to settle for a more modest 
objective. More specifically, we are interested in functions f : A(B) — R that are attainable with 
respect to a given constraint set T. As a target we suggest the closed convex hull of the reward-in- 
hindsight, r*. After defining it, we prove that it is indeed attainable. In the next section, we will also 
show that it is the highest possible attainable function, when there is a single constraint. 

Given a function f : X +> R, over a convex domain X, its closed convex hull is the function 
whose epigraph is 


conv({(x,r):r > f(x)}), 


where conv(D) is the convex hull, and D is the closure of a set D. We denote the closed convex hull 
of r* by r°. 

We will make use of the following facts. Forming the convex hull and then the closure results 
in a larger epigraph, hence a smaller function. In particular, r°(q) < r*(q), for all q. Furthermore, 
the closed convex hull is guaranteed to be continuous on A(B). (This would not be true if we had 
considered the convex hull, without forming its closure.) Finally, for every q in the interior of A(B), 
we have: 


rq) = inf X air“ (qi) (8) 


q1:42,--4kKEA(B),01 ;---,0k Fy 


k 
s.t. )aigi(b)=q(b), VbEB, 
i=l 


a;>0, i=1,2,...,k, 


k 
$o; =, 
i=l 
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where k can be taken equal to |B| + 2 by Caratheodory’s Theorem. 

The following result is proved using Blackwell’s approachability theory. The technique is simi- 
lar to that used in other no-regret proofs (e.g., Blackwell, 1956b; Mannor and Shimkin, 2003), and 
is based on the convexity of a target set in an appropriately defined space. 


Theorem 5 Let Assumption 1 hold for a given convex set T C R¢. Then r° is attainable with respect 
to T. 


Proof Define the following game with vector-valued payoffs, where the payoffs belong to R x 
R? x A(B) (a |B| +d +1 dimensional space, which we denote by M). Suppose that P1 plays ar, 
P2 plays b,, P1 obtains an immediate reward of r, and an immediate penalty vector of c+. Then, the 
vector-valued payoff obtained by P1 is 


m, = (r:,cr,e(D:)) , 


where e(b) is a vector of zeroes, except for a 1 in its bth component. It follows that the average 
vector-valued reward at time t, which we define as m, = ye | Mz, satisfies: fuy = (F:,€+,G), where 
F, C;, and G, were defined in Eqs. (1), (2), and (4), respectively. Consider the sets: 


B=—1{neg) eM irr (q)}, B={(r,c,q)EM:ceET}, 


and let B = Bı 1B). Note that B is a convex set. We claim that B is approachable. Let m: 
A(A) x A(B) — M describe the expected payoff in a single stage game, when P1 and P2 choose 
actions p and q, respectively. That is, 


m(p.4) = (Ypla)al)R(a.b), Yrla)a(b)C(a,b), a). 


a,b 


Using the sufficient condition for approachability of convex sets (Blackwell, 1956a), it suffices to 
show that for every q there exists a p such that m(p,q) € B. Fix q € A(B). By Assumption 1, the con- 
straint X4 p p(a)q(b)C(a,b) €T is feasible, which implies that the program (5) has an optimal solu- 
tion p*. It follows that m(p*,q) € B. We now claim that a strategy that approaches B also attains r° 
in the sense of Definition 1. Indeed, since B C B we have that Pr(d(c,, T) > € infinitely often) = 0 
for every £ > 0. Since B C B, and using the continuity of r°, we obtain liminf (7, —r°(G,)) > 0. E 


We note that Theorem 5 is not constructive. Indeed, a strategy that approaches B, based on a 
naive implementation Blackwell’s approachability theory, requires an efficient procedure for com- 
puting the closest point in B,and therefore a computationally efficient description of B, which may 
not be available (we do not know whether B can be described efficiently). This motivates the devel- 
opment of the calibration based scheme in Section 6. 


Remark 6 Convergence rate results also follow from general approachability theory, and are gen- 
erally of the order of t~'/3; see Mertens et al. (1994). It may be possible, perhaps, to improve upon 


this rate and obtain t~'/?, which is the best possible convergence rate for the unconstrained case. 


Remark 7 For every q € A(B), we have r*(q) > v, which implies that r°(q) > v. Thus, attaining r° 
guarantees an average reward at least as high as the value of the game. 
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4.1 Degenerate Cases 


In this section, we consider the degenerate cases where the penalty vector is affected by only one of 
the players. We start with the case where P1 alone affects the penalty vector, and then discuss the 
case where P2 alone affects the penalty vector. 

If P1 alone affects the penalty vector, that is, if C(a,b) = C(a,b’) for alla € A and b,b’ € B, then 
r*(q) is convex. Indeed, in this case, Eq. (5) becomes (writing C(a) for C(a,b)) 


ri(q)= max a)q(b)R(a,b), 
(9 peas) caer”! dal ) ( ) 


which is the maximum of a collection of linear functions of q (one function for each feasible p), and 
is therefore convex. 

If P2 alone affects the penalty vector, that is, if c(a,b) = c(a',b) for all b € B anda,a' € A, then 
Assumption 1 implies that the constraint is always satisfied. Therefore, 


r*(q)= max Pla) a,b), 


eA 


which is again a maximum of linear functions, hence convex. 
We conclude that in both degenerate cases, if Assumption | holds, then the reward-in-hindsight 
is attainable. 


5. Tightness of the Convex Hull 


We now show that r° is the maximal attainable function, for the case of a single constraint. 


Theorem 8 Suppose that d = 1, T is of the form T = {c | c < co}, where co is a given scalar, and 
that Assumption 1 is satisfied. Let 7: A(B) +> R be a continuous attainable function with respect to 
the scalar co. Then, r°(q) > F(q) for all q € A(B). 


Proof The proof is constructive, as it provides a concrete strategy for P2 that prevents P1 from 
attaining 7, unless r°(q) > 7(q) for every q. Assume, in order to derive a contradiction, that there 
exists some 7 that violates the theorem. Since 7 and r° are continuous, there exists some q? € A(B) 
and some € > 0 such that #(q) > r°(q) +£ for all q in an open neighborhood of q?. In particular, q? 
can be taken to lie in the interior of A(B). Using Eq. (8), it follows that there exist q!,...,q* € A(B) 
and 0t;,..., 0% (with k < |B| +2, due to Caratheodory’s Theorem) such that 


Lori) <r°(g) += <#(q) — =; 
=] 2. 2 


k 
P aq (b)=(b), VoEeB, Ya=1; «20, Vi. 
i=] i— 


Let t be a large positive integer (t is to be chosen large enough to ensure that the events of 
interest occur with high probability, etc.). We will show that if P2 plays each q’ for [a;t] time 
steps, in an appropriate order, then either P1 does not satisfy the constraint along the way or ĉr < 


F(Gx) —€/2. 
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Reward 
Reward 


p'Cq=c 























Figure 2: In either part (a) or (b) of the figure, we fix some g € A(B). The triangle is the set of 
possible reward-cost pairs, as we vary p over the set A(A). Then, for a given value c in 
the upper bound on the cost (cf. 10), the shaded region is the set of reward-cost pairs that 
also satisfy the cost constraint. 


We let g', i= 1,...,k, be fixed, as above, and define a function f; : R? — RU {—co} as: 


file) = ae ees (b)R(a,b), (9) 
subject to $ p(a)q'(b)C(a,b) < c, (10) 
a,b 


where the maximum over an empty set is defined to equal —co. Observe that the feasible set (and 
hence, optimal value) of the above linear program depends on c. Figure 2 illustrates how the feasible 
sets to (10) may depend on the value of c. By viewing Eqs. (9)-(10) as a parametric linear program, 
with a varying right-hand side parameter c, we see that f;(c) is piecewise linear, concave, and 
nondecreasing in c (Bertsimas and Tsitsiklis, 1997). Furthermore, f;(co) = r*(q'). Let 3f? be 
the right directional derivative of f; at c = co, and note that ð f > 0. From now on, we assume 
that the q! have been ordered so that the sequence ð ft is nonincreasing (e.g., as in Figure 3). 
To visualize the ordering that we have introduced, consider the set of possible pairs (7,c), given 
a fixed q. That is, consider the set M(q') = {(r,c) : dp € A(A)s.t. r = Za p p(a)q'(b)R(a,b), c = 
Yap P(a)q'(b)C(a,b)}. The set M(q') is the image of the simplex under a linear transformation, and 
is therefore a polytope, as illustrated by the triangular areas in Figure 2. The strategy of P2 is to first 
play q’ such that the p that maximizes the reward (Eq. 9) satisfies Eq. (10) with equality. (Such a 
qi results in a set M(q') like the one shown in Figure 2(b).) After all these qf are played, P2 plays 
those q! for which the p that maximizes the reward (Eq. 9) satisfies Eq. (10) with strict inequality, 
and ðf; =0. (Such a g' results in a set M(q') like the one shown in Figure 2(a).) 

Suppose that P1 knows the sequence q!,...,q* (ordered as above) in advance, and that P2 fol- 
lows the strategy described earlier. We assume that Tt is large enough so that we can ignore the 
effects of dealing with a finite sample. Let p be the average of the mixed actions chosen by P1 
while player P2 plays q'. We introduce the constraints 





l 
ya Y pi(a)g'(b)C(a,b) < co} ai, L=1,2,...,k. 


i=l a,b i=1 
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Figure 3: An example of functions f; ordered according to ð f. 


These constraints must be satisfied in order to guarantee that ¢; has negligible probability of sub- 
stantially exceeding co, at the “switching” times from one mixed action to another. If P1 exploits 
the knowledge of P2’s strategy to maximize her average reward at time qT, the resulting expected 
average reward at time T will be the optimal value of the objective function in the following linear 
programming problem: 


k 
net Pea Ra, b) 
ee j=) a,b 
£ f ; £ 
at.) o} pag (b)C(a,b) < co} Oy £=1,2,..k, (11) 
= a,b i=1 


i=1 


p’e€A(A), &€=1,2,...,k. 


= 


Of course, given the value of X, p'(a)q'(b)C(a,b), to be denoted by c;, player P1 should choose 
a p' that maximizes rewards, resulting in X4 p p'(a)q'(b)R(a,b) = fi(ci). Thus, the above problem 
can be rewritten as 


max J a%ifi(ci) 
l 4 
s.t. aici < co), a, £1252 5k (12) 


i=l i=l 


We claim that letting c; = co, for all i, is an optimal solution to the problem (12). This will then 
imply that the optimal value of the objective function for the problem (11) is yh Oi f;(co), which 
equals Z; ajr*(q'), which in turn, is bounded above by #(q°) —€/2. Thus, ĉr <#(q°) —€/2+8(t), 
where the term (Tt) incorporates the effects due to the randomness in the process. By repeating 
this argument with ever increasing values of t (so that the stochastic term 6(t) is averaged out 
and becomes negligible), we obtain that the event 7, < #(q°) —€/2 will occur infinitely often, and 
therefore 7 is not attainable. 

It remains to establish the claimed optimality of (co,...,co). Suppose that (¢1,...,¢,) Æ 
(co,---,Co) is an optimal solution of the problem (12). If ¢; < co for all i, the monotonicity of 
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the f; implies that (co,...,co) is also an optimal solution. Otherwise, let j be the smallest index for 
which ¢; > co. If of F = (0 (as in the case shown in Figure 2(b)) we have that f;(c) is maximized at 
co for all i > j and (co,...,co) is optimal. Suppose that ð fF > 0. In order for the constraint (12) 
to be satisfied, there must exist some index s < j such that Cs < co. Let us perturb this solution 
by setting 6 = min{0,(co — Cs), @;(€; — co) }, increasing Cs to Čs = Cs + /Os, and decreasing Cj to 
č; =t; —6/a;. This new solution is clearly feasible. Leto fy =limg,o(fs(co) — fs(co —€))/€, which 
is the left derivative of fẹ at co. Using the concavity of f;, and the earlier introduced ordering, we 
have df, >of," > ðf. Observe that 


Aes) = fee) 07, 0/0, 
Fi(Ej) = fE) — Of; S/a, 


so that Os f5(E;) +a; fj(Ej) > Os fs(Es) +a; f;(E;). Therefore, the new solution must also be optimal, 
but has fewer components that differ from cp. By repeating this process, we eventually conclude 
that (co,...,Co) is an optimal solution of (12). E 


To the best of our knowledge, this is the first tightness result for a performance envelope (the 
reward-in-hindsight) different than the Bayes envelope, for repeated games. On the other hand, we 
note that our proof relies crucially on the assumption of a single constraint (d = 1), which allows us 
to order the ðf”. 


6. Attaining the Convex Hull Using Calibrated Forecasts 


In this section, we consider a specific strategy that attains the convex hull, thus providing a con- 
structive proof for Theorem 5. The strategy is based on forecasting P2’s action, and playing a best 
response (in the sense of Eq. 5) against the forecast. The quality of the resulting strategy depends, 
of course, on the quality of the forecasts; it is well known that calibrated forecasts lead to no-regret 
strategies in standard repeated matrix games. See Foster and Vohra (1997) and Cesa-Bianchi and 
Lugosi (2006) for a discussion of calibration and its implications in learning in games. In this 
section we consider the consequences of calibrated play for repeated games with constraints. 

We start with a formal definition of calibrated forecasts and calibrated play, and then show that 
calibrated play attains r“ in the sense of Definition 1. 

A forecasting scheme specifies at each stage k a probabilistic forecast qg € A(B) of P2’s action 
bg. More precisely a (randomized) forecasting scheme is a sequence of maps that associate with 
each possible history hg—ı during the first k — 1 stages a probability measure ug over A(B). The 
forecast qg € A(B) is then selected at random according to the distribution ug. Let us clarify that for 
the purposes of this section, the history is defined to include the realized past forecasts. 

We shall use the following definition of calibrated forecasts. 


Definition 9 (Calibrated forecasts) A forecasting scheme is calibrated if for every (Borel measur- 
able) set Q C A(B) and every strategy of P1 and P2 


_ 1, 
lim 7 $ {qr € O}(e(br)— Gr) =0, a.s., (13) 
t=1 
where e(b) is a vector of zeroes, except for a 1 in its bth component. 
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Calibrated forecasts, as defined above, have been introduced into game theory in Foster and Vohra 
(1997), and several algorithms have been devised to achieve them (see Cesa-Bianchi and Lugosi, 
2006, and references therein). These algorithms typically start with predictions that are restricted to 
a finite grid, and gradually increase the number of grid points. 

The proposed strategy is to let P1 play a best response against P2’s forecasted play while still 
satisfying the constraints (in expectation, for the single stage game). Formally, we let: 


p*(q) = argmax} p(a)q(b)R(a,b) 
pEA(A) a,b 


s.t. 2 plaja) (a,b) ET, 


where in the case of a non-unique maximum we assume that p* (q) is uniquely determined by some 
tie-breaking rule; this is easily done, while keeping p*(-) a measurable function. The strategy is 
to play p; = p*(q:), where q: is a calibrated forecast of P2’s actions.* We call such a strategy a 
calibrated strategy. 

The following theorem states that a calibrated strategy attains the convex hull. 


Theorem 10 Let Assumption I hold, and suppose that P1 uses a calibrated strategy. Then, r° is 
attained with respect to T. 


Proof Fix £ > 0. We need to show that by playing the calibrated strategy, P1 obtains liminf,_,.. (7; — 
r°(G:)) > 0 and lim sup, dist(¢;, T) < 0, almost surely. 

Fix some € > 0. Consider a partition of the simplex A(B) to finitely many measurable sets 
Q1,Q2,...,Q¢ such that q,q' € Q; implies that ||g — q'|| < £ and ||p* (q4) — p*(q’)|| < £. (Such a 
partition exists by the compactness of A(B) and A(A). The measurability of the sets Q; can be 
guaranteed because the mapping p*(-) is measurable.) For each i, let us fix a representative element 
q' € Qj, and let p' = p*(q'). 

Since we have a calibrated forecast, Eq. (13) holds for every Q;, 1 < i < £. Define T;(i) = 
£t l{qr: € Qi} and assume without loss of generality that T, (i) > 0 for large t (otherwise, eliminate 
those i for which I’,(i) = 0 for all t£, and renumber the Q;). To simplify the presentation, we assume 
that for every i, and for large enough ft, we have T’;(i) > et. (If for some i, and t this condition is 
violated, the contribution of such an i in the expressions that follow will be O(e).) 

By a law of large numbers for martingales, we have 


1 t 
li ĉ—-— ) Claz,b =0, S. 
=A (a D (ar, o) i 
By definition, we have 


: oh 
LCb) =} 7 2 Cla, b)z 


a,b t 








~| = 


È Mae € ae =a {b=} 


Observe that whenever q € Qi, we have ||P: -= r'| < £, where pr = p* (qz) and p! = p*(q') because 
of the way the sets Q; were constructed. By martingale convergence, the frequency with which a 





4. When the forecast u, is mixed, qr is the realization of the mixed rule. 
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will be selected whenever gz € Q; and b; = b, will be approximately p’ (a). Hence, for all b, 


z E Ha € Oi} {br =b} <e, 
l tl 





lim sup 


t—oo 


T; 





i g i 
Eg È Hae oMa =a} {be = b} plo 


almost surely. By the calibration property (13) for Q = Q;, and the fact that whenever q,q' € Q;, we 
have ||q —q'|| < £, we obtain 











I ; 
limsup|—~ } 1{qr € Oi} 1{b, =b} — q'(b)| <e, a.s. 
f—+ 0 I; (i) t=1 
By combining the above bounds, we obtain 
RNI T;(i) ifani 
lim Cr -y —— Yecla.b)p (a)q'(b)| < 2e, a.s. (14) 
7 i a,b 








Note that the sum over index i in Eq. (14) is a convex combination (because the coefficients 
T;(i)/t sum to 1) of elements of T (because of the definition of p’), and is therefore an element of 
T (because T is convex). This establishes that the constraint is asymptotically satisfied within O(e). 
Note that in this argument, whenever I, (i) /t < £, the summand corresponding to i is indeed of order 
O(e) and can be safely ignored, as stated earlier. 

Regarding the average reward, an argument similar to the above yields 


Ty (i ae 

liminfî, > liminf Y ri) y'R(a,b) p'(a)q'(b) —2e, a.s. 
ae R i a,b 

Next, observe that 


L Ë ERub pado- 


i a,b 


I; 








D eigi) > (ya, 


t 


where the equality is a consequence of the definition of p’, and the inequality follows by the def- 
inition of r° as the closed convex hull of r*. Observe also that the calibration property (13), with 
Q = A(B), implies that 


1 t 
lim ||G; — — =(), a.s. 
t— oo dr f La 














In turn, since ||g; —q'|| < £ for a fraction I, (i)/t of the time, 


te T;(i) ; 
ae ee 
t=1 


i 





A r i i 
D 


i 


lim sup 


t— oo 


<E, a.s. 


























= lim sup 
t— o 
Recall that the function r° is continuous, hence uniformly continuous. Thus, there exists some 


function g, with limęjog(€) = 0, such that when the argument of r° changes by at most £, the value 
of r° changes by at most g(€). By combining the preceding results, we obtain 


liminf?; > r°(qz) —2€— g(€), a.s. 
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The above argument involves a fixed £, and a fixed number £ of sets Q;, and lets t increase to in- 
finity. As such, it establishes that for any € > 0 the function r° — 2e — g(€) is attainable with respect 
to the set T? defined by T? = {x | dist(x,7) < 2e}. Since this is true for every € > 0, we conclude 
that the calibrated strategy attains r“ as claimed. = 


7. Algorithms 


The results in the previous section motivate us to develop algorithms for online learning with con- 
straints, perhaps based on calibrated forecasts. For practical reasons, we are interested in computa- 
tionally efficient methods, but there are no known computationally efficient calibrated forecasting 
algorithms. For this reason, we will consider related heuristics that are similar in spirit, even if they 
do not have all the desired guarantees. 

We first consider a method based on the weighted average predictor. The algorithm in Table 1 
keeps track of the performance of the different actions in the set A, updating a corresponding set 
of weights accordingly at each step. The main idea is to quantify “performance” by a linear com- 
bination of the total reward and the magnitude of the constraint violation. The parameter À > 0 of 
the algorithm, which acts similar to a Lagrange multiplier, determines the tradeoff between these 
two objectives. When the average penalty is higher than co (i.e., there is a violation), the weight 
of the cost term increases. When the average penalty is lower than co, the weight of the cost term 
decreases. The parameters M and M are used to bound the magnitude of the weight of the cost term; 
in the experiments reported in Section 8, they were set to 1000 and 0.001, respectively. 





1. Set A, wo, M, and M. 
2. Fort =1,2,...: 


(a) Sample an independent random variable a, distributed so that 


=e. sips 
acA Wr a) 





fora eA. 
(b) Compute: 
w,(a) = w,- (a) exp (n (R(a,b;) —AC(a,b;))), acA. 


(c) Fort = 1,2,..., update A: 








_ f min(2à,M), ifô,>co, 
~ | max(à/2,M), otherwise. 





Table 1: Exponentially weighted average predictor. 


The second algorithm uses the tracking forecaster (Mannor et al., 2007) as the forecasting 
method. This forecaster predicts that the distribution of the next action as a weighted average of 
previous actions, weighing recent actions more than less recent ones others. For the special case of 
only two actions, it is calibrated, but not calibrated in general. There are, however, some special 
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cases where it is calibrated, in particular if the sequence it tries to calibrate comes from a source 
with some specific properties; see Mannor et al. (2007) for details. The algorithm is presented in 
Table 2. If there is a current violation, it selects an action that minimizes the immediate forecasted 
cost. If the current average penalty does not violate the constraint, it selects a best response to the 
forecasted action of P2, while satisfying the constraints. 





1. Set p € (0,1), co, and fo = (1/|BI)T. 
2. Fort = 1,2,...: 


(a) If t = 1 or ĉ > cg, choose an action that minimizes the worst-case cost: 


a, € argmin (C(a,b) f;-1(b)), 


acA 


(b) Otherwise (if ĉs < co and t > 1), solve 


w 2 pla) R(a,b) fr_-1(b), 


pEA(A 


subject to Fpl) a)C(a,b) fr_1(b) < co. 
a,b 


and choose a random action distributed according to the solution to the above linear 
program. 


(c) After observing b;, update the forecast f; on the probability distribution of the next 
opponent action b;41: 


F= fi- + (1/4) (es, — fi—1), 


where ep is a unit vector in R!” with the element 1 in the component corresponding 
tobe B. 











Table 2: Tracking forecaster. 


8. Experimental Setup 


Our experiment addresses the problem of minimizing power consumption in a computer with a hu- 
man user. The agent is a low-level software controller that decides when to put the central processor 
(CPU) into a low-power state, thereby reducing power expenditures during periods when the user 
is idle. The system is driven by a human user, as well as different hardware processes, and can 
be realistically assumed to be non-stationary. The actions of the system correspond to hardware 
interrupts (most interrupts are generated by hardware controllers on the motherboard such as direct 
memory access, hard disk interrupts and networking interrupts) and the ongoing running processes. 
In the particular application at hand, there is a software interrupt (generated by the Windows oper- 
ating system) every 16 milliseconds. The times of these interrupts are the decision epochs, at which 
the software controller can decide if and when to put the CPU to sleep before the next scheduled 
periodic interrupt. 
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However, saving energy by putting the processor in the low-power state comes at a cost. In the 
low-power state, a delay is incurred each time that the processor moves back into the high-power 
state in response to user-generated interrupts. We wish to limit the delay perceived by the human 
user. For this purpose, we assign a cost to the event that an interrupt arrives while the processor is 
in the low-power state, and impose a constraint on the time average of these costs. A similar model 
was used in Kveton et al. (2008), and we refer the reader to that work for further details. 

We formulate the problem as follows. We divide a typical 16 millisecond interval into ten 
intervals. We let P1’s action set be A = {0,0.1,0.2,...,1}, where action a corresponds to turning 
off the CPU after 16a milliseconds (the action a = 1 means the CPU is not turned off during the 
interval while the action a = 0 means it is turned off for the whole interval). Similarly, the action set 
of P2 is B= {0,0.1,0.2,...,0.9}, where action b corresponds to an interrupt after 16b milliseconds. 
(Note that the action b = 0 means there is no interrupt and that there is no point in including an 
action b = 1 in B since it would coincide with the known periodic interrupt.) The assumption is that 
an interrupt is handled instantaneously so if the CPU chooses a slightly larger than b it maximizes 
the power savings while incurring no penalty for observed delay (it is assumed for the sake of 
discussion that only a single interrupt is possible in each 16 millisecond interval). We define the 
reward at each stage as follows: 


l—a, ifb=Oora>b, _ that is, if no interrupt occurs or an interrupt occurs 
before the CPU turns off, 

b—a, ifb>Oanda<b, _ that is, if there is an interrupt 
after the CPU is turned off. 


The cost is: 
1, ifa<bandb>0O, 
C(a,b) = 


0, otherwise. 


In “normal” operation where the CPU is powered throughout, the action is a = 1 and in that case 
there is no reward (no power saving) and no cost (no perceived delay). When a = 0 the CPU is 
turned off immediately and in this case the reward will be proportional to the amount of time until 
an interrupt (or until the next decision). The cost in the case a = 0 is 0 only is there is no interrupt 
(b=0). 

We used the real data trace obtained from what is known as MobileMark 2005 (MMO05), a 
performance benchmark that simulates the activity of an average Microsoft Windows user. This 
CPU activity trace is 90 minutes long and contains more than 500,000 interrupts, including the 
periodic scheduled interrupts mentioned earlier. The exponentially weighted algorithm (Table 1) 
and the tracking forecaster (Table 2) were run on this data set. Figure 4 shows the performance of 
the two algorithms. The straight line shows the tradeoff between constraint violation and average 
reward by picking a fixed action over the entire time horizon. The different points for the exponential 
weighted predictor (Table 1) or the tracking forecaster (Table 2) correspond to different values of 
co. We observe that for the same average cost, the tracking forecast performs better (i.e., gets higher 
reward). 

We selected co = 0.3 and used both algorithms for the MMOS trace. Figures 5(a) and 5(b) show 
the instantaneous cost incurred by the tracking forecaster and the weighted average forecaster over 
the same short period. It should be observed that the cost of the algorithms is different, reflecting 
the fact that different policies are employed. Figures 6(a) and 6(b) show the time evolution of the 
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Figure 4: Plot of average reward against constraint violation frequency from experiments in power 


Instantaneous cost 


Figure 5: Instantaneous cost incurred by the tracking forecaster and weighted average predictor 
with target constraint co = 0.3 for the MMOS data. 


average reward and average cost for the same experiment. In spite of not being calibrated, the 
tracking forecast based algorithm outperforms the exponentially weighted based algorithm. 
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(a) Tracking forecaster 


9, Conclusions 


There are several open problems and directions for future research that are worth mentioning. First, 
the issue of convergence rate is yet to be settled. We noted that there exists an algorithm based on 
approachability that converges at the rate of t—'/3, and that the usual lower bound of t~!/? holds. 
The other algorithm based on calibration suffers from potentially even worse convergence rate, as 
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(b) Weighted average predictor 
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Figure 6: Time evolution of average reward and average cost for the tracking forecaster and 
weighted average forecaster with co = 0.3 for the MMOS data. 


we are not aware of any approximate calibration algorithm with comparable convergence rates. 
Second, the complexity of these two online learning algorithms leaves much to be desired. The 
complexity of a policy based on approachability theory is left undetermined because we do not 
have a specific procedure for computing P1’s action at each stage. The per stage complexity is 
unknown for calibrated forecasts, but is exponential for approximately calibrated schemes (Cesa- 
Bianchi and Lugosi, 2006). Moreover, it is not clear whether online learning with constraints is as 
hard computationally as finding a calibrated forecast. Third, we only established the tightness of 
the lower convex hull of the Bayes envelope for the case of a one-dimensional penalty function. 
This is a remarkable result because it establishes the tightness of an envelope other than the Bayes 
envelope, and we are not aware of any such results for similar settings. However, it is not clear 
whether such a result also holds for two-dimensional penalties. In particular, the proof technique of 
the tightness result does not seem to extend to higher dimensions. 


Our formulation of the learning problem (learning with pathwise constraints) is only a first 
step in considering multi-objective problems in online learning. In particular, other formulations, 
for example, that consider the number of time-windows where the constraints are violated, are of 
interest; see Kveton et al. (2008). 
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